Repository navigation
Cap shard scale-up by indexing CPU capacity - #7007
Draft
fulmicoton-dd wants to merge 1 commit into
Draft
fulmicoton-dd wants to merge 1 commit into
fulmicoton-dd wants to merge 1 commit into
Conversation
Under sudden load, the control plane keeps opening shards, and the scheduler keeps creating pipelines for them, beyond what indexers can process. When `QW_ENABLE_INDEXING_CAPACITY_SHARD_CAP` is set, throughput-driven shard scale-up is clamped so that the pipelines implied by all shards do not exceed the CPU capacity advertised by indexers divided by `QW_CPU_MILLIS_PER_PIPELINE` (default: 1000, one pipeline per core). Shards per pipeline are computed with the scheduler's load model. Scale-ups needed to reach `min_shards` and the first shards of a source are never clamped. A new gauge `shard_scale_up_at_max_capacity` is set to 1 when a scale-up gets clamped, and 0 otherwise.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Under sudden load, the control plane keeps scaling up shards, and the indexing scheduler keeps creating pipelines for them. Past a point, this does not increase indexing throughput: indexers are saturated.
Change
Behind
QW_ENABLE_INDEXING_CAPACITY_SHARD_CAP(off by default), the ingest controller clamps throughput-driven shard scale-up:QW_CPU_MILLIS_PER_PIPELINE(default1000, one pipeline per core).ceil(num_shards / max_shards_per_pipeline). Open and closed shards are counted.max_shards_per_pipelineuses the scheduler's load model.max_shards_per_pipelineslots per pipeline left in the budget.Scale-ups needed to reach
min_shards, and the first shards of a source, are never clamped.New gauge
quickwit_control_plane_shard_scale_up_at_max_capacity:1when the last scale-up was clamped,0otherwise.Docs:
docs/internals/ingest-v2.md.Notes
QW_CPU_MILLIS_PER_PIPELINEto ~3000–3200 matches it.Tests
ingest/indexing_capacity_cap.rs(budget, pipelines used, clamp).ingest_controller.rs: end-to-end clamp throughhandle_local_shards_update, no clamping belowmin_shards, no clamping when the cap is disabled.cargo nextest run -p quickwit-control-plane: 158 passed.🤖 Generated with Claude Code